1. Field of the Invention
The present disclosure relates to imaging systems and methods that include a de-noising process. In particular, the disclosure relates to systems and methods that use a near infrared image to guide the de-noising process.
2. Description of the Related Art
It can be challenging to take high quality photographs and videos under low light conditions. Without sufficient ambient light, photographers generally have four options to increase picture quality. The first option is to use a high ISO when capturing the image. Increasing sensor gain can effectively increase signal strength to get bright images within short exposure time. However, the image noise is inevitably increased as well, thus the signal-to-noise ratio (SNR) is not improved.
A second option is to capture the image using a large aperture. Allowing more light pass through camera lenses is a very straightforward way to improve image quality, however changing aperture size will also affect depth of field. Further, the effect is very limited when sensor and lens have been built to accommodate a small form factor, e.g. cell phone cameras.
A third option is to use long exposure time to capture the image. Extended exposure time can increase SNR, but may increase undesired motion blur in the captured image.
A fourth option is to employ flash or other strong artificial light to the scene in order to obtain a sharp, noise-free image. However, as the color temperature and intensity of flash light are usually quite different from those of the ambient light, the use of flash may ruin ambiance atmosphere and introduce unwanted artifacts, such as red eye, undesired reflections, harsh shadows, and intense light highlights and reflections.
Out of above four options, photographers usually prefer to use high ISO images and apply noise reduction to the captured image. Image de-noising is an intensively studied problem and numerous methods exist. However, even with the state-of-the-art image de-noising methods, it is still very difficult to obtain a high quality noise-free photo, especially when noise level is high.
Conventional single-image de-noising solutions consist of several different methods. Image filtering based methods selectively smooth parts of a noisy image. Wavelet-based methods rely on the careful shrinkage of wavelet coefficients. Image prior based methods learn a dictionary from noise-free images and de-noise images by approximating them using a sparse linear combination of the elements in the dictionary. More recent approaches exploit the “non-local” property of natural images: small patches in natural images tend to repeat themselves within the image. In the last class of methods, BM3D well represents the state-of-the-art in single image de-noising. However, a common fundamental problem with single-image de-noising approaches is that they are not able to distinguish between noise and original image signals, especially with respect to the finer image details. Hence, those approaches generate reasonably good-quality results for images with relatively low noise levels, but generally produce over-smoothened results with many artifacts for images containing high noise levels.
In an attempt to overcome the limitations of the single-image de-noising approaches described above, dual-image methods introduce another image to guide the de-noising process. Two images are captured of the same scene. Image filtering of the first image using the guidance of the second image is then applied to better preserve image structure, and image detail transfer may be applied to enhance fine image details. For example, the guidance image may be captured under different lighting conditions than the first image, and therefore may contain a different level of detail of the image scene. The additional details in the guidance image may be used to enhance the quality of the first image. The first type of dual-image de-noising methods uses a visible flash image as the guidance image. However, this method can easily blur weak edges and introduce artifacts from the guidance image. Additionally, the visible flash is intrusive to use under low-light conditions, and is even prohibited in certain environments.
Most recently, a second type of dual-image de-noising method has emerged which uses an “invisible flash,” for example a flash using near infrared or ultra violet light, in order to capture an image outside of the visible light band, and use that “dark flash” image to help denoise a corresponding visible light image. Gradient transfer via a simple Poisson equation can be adopted for de-noising the visible light image using the dark flash image as a reference. Such methods are able to perform de-noising and detail transfer simultaneously and sometimes achieve high quality de-noising results. However, the gradient constraint from the dark flash image may be too heuristic and not well adapted to handle the differences between the visible light image and the dark flash image. This results in a noticeable appearance change in the de-noising results, especially when handling images with intensive noise.